feat!: upgrade DataFusion to 55 - #8555
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request upgrades the DataFusion dependency from version 53.1.0 to 54.0.0, which involves extensive refactoring to adapt to API changes such as the removal of explicit as_any() downcasting, updates to the Cast expression structure, and changes to PruningStatistics and partition_statistics signatures. It also introduces a frozen hash_v1 module to preserve compatibility with legacy count_hash states. The review identified critical compilation errors: a missing import for as_largestring_array in the new hash_v1.rs file, and incorrect iterations over the return value of hash_utils::create_hashes in union_distinct_on.rs since it now returns Result<()> instead of the hashes.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
github-actions
Bot
added
docs-not-required
|
orc and pg-catalog are our managed libraries, make sure it merges to upstream before merging this. |
|
I'm working on the pg-catalog upgrade for datafusion 54. will commit to this branch. |
|
Thanks — I see |
|
DCO is a non-blocking check, just set it passed. |
|
Thanks, DCO is green now. One remaining managed-dependency setup: |
|
datafusion-orc is now on 0.9 |
|
The pg-catalog 0.18 integration uncovered one remaining upstream compatibility gap during real PostgreSQL-protocol SQLness validation: OID aliases are represented as |
|
Updated locally to the managed |
|
Could you confirm the intended ownership for the DataFusion thin-fork pin itself? The existing dependency used |
|
The managed DataFusion 54 path is now available. I created |
|
datafusion 54.1 is out. |
ff94fb3 to
fd09114
Compare
fd09114 to
cf9188e
Compare
Address review feedback on PR GreptimeTeam#8555: extract duplicated coordinate x resolution signature construction, coordinate/resolution argument parsing, and per-row resolution conversion into shared helpers (signature_of_coordinates_and_resolution, extract_coordinate_resolution_args, resolution_at). Drop redundant alias statics (RESOLUTION_TYPES, DISTANCE_TYPES, POSITION_TYPES) in favor of INTEGER_TYPES. Behavior, signatures, error messages, and visibility are unchanged. Signed-off-by: discord9 <discord9@163.com>
|
Addressed: |
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Merge main at 38577c8 without rewriting the existing PR history. Retain last-row scan hints and dynamic filtering while adapting new main downcasts to DataFusion 55. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
User-visible behavior changes inherited from the upgradeMost of these are not introduced by this PR's own code, but they ship with it and are visible to users. They should be listed in the release notes, and the PR checklist needs updating —
The |
killme2008
left a comment
killme2008
left a comment
There was a problem hiding this comment.
A few things from reading the diff. Breaking changes are in a separate comment above.
One more that has no source line: the description says the current head is e749334f0d4, but the head of this PR is 3b8c0a85b5. None of the validation results in the description refer to the commit being reviewed.
| | data/greptime/public/<TABLE_ID>/ | data/greptime/public/<TABLE_ID>/<REGION_ID>_<REGION_NUMBER>/index/<UUID>.puffin |<NUM>|<NUM>|<NUM>|<NUM>|<NUM>| <UUID> |<NUM>| bloom_filter | column |<NUM>| {"column":1} |<NUM>| {"bloom":{"rows_per_segment":10240,"segment_count":1,"row_count":1,"bloom_filter_size":64}} |<NUM>| | ||
| | data/greptime/public/<TABLE_ID>/ | data/greptime/public/<TABLE_ID>/<REGION_ID>_<REGION_NUMBER>/index/<UUID>.puffin |<NUM>|<NUM>|<NUM>|<NUM>|<NUM>| <UUID> |<NUM>| bloom_filter | column |<NUM>| {"column":1} |<NUM>| {"bloom":{"rows_per_segment":10240,"segment_count":1,"row_count":1,"bloom_filter_size":64}} |<NUM>| | ||
| | data/greptime/public/<TABLE_ID>/ | data/greptime/public/<TABLE_ID>/<REGION_ID>_<REGION_NUMBER>/index/<UUID>.puffin |<NUM>|<NUM>|<NUM>|<NUM>|<NUM>| <UUID> |<NUM>| bloom_filter | column |<NUM>| {"column":1} |<NUM>| {"bloom":{"rows_per_segment":10240,"segment_count":1,"row_count":1,"bloom_filter_size":64}} |<NUM>| | ||
| | data/greptime/public/<TABLE_ID>/ | data/greptime/public/<TABLE_ID>/<REGION_ID>_<REGION_NUMBER>/index/<UUID>.puffin |<NUM>|<NUM>|<NUM>|<NUM>|<NUM>| <UUID> |<NUM>| inverted | column |<NUM>| {"column":0} |<NUM>| {"inverted":{"bitmap_type":"Roaring","base_offset":0,"inverted_index_size":81,"relative_fst_offset":26,"fst_size":55,"relative_null_bitmap_offset":0,"null_bitmap_size":8,"segment_row_count":1024,"total_row_count":1}} |<NUM>| |
There was a problem hiding this comment.
The meta_json keys went from sorted to source order here, and puffin_index.rs isn't touched in this PR.
The cause is substrait 0.62.2 -> 0.63.0. 0.63.0 declares serde_json = { version = "1.0.145", features = ["preserve_order"] }, and feature unification turns that on for the whole workspace, so serde_json::Map becomes an IndexMap everywhere. Cargo.lock confirms it: serde_json on main has no indexmap dependency, on this branch it does.
So this affects every JSON we build or round-trip through serde_json::Value, not just this column. JSON column values are fine, they go through jsonb::parse_value. Please call it out in the description, and check that nothing compares or hashes serialized JSON bytes.
There was a problem hiding this comment.
Updated in bc7ecdc. We decided not to accept global serde_json ordering changes as part of this upgrade. The production feature graph revealed two enablers, not just Substrait: substrait 0.63 and datafusion-physical-plan.
Both are now scoped locally:
- GreptimeTeam/substrait-rs@91ec978 preserves named-struct field order/ordinals with an explicit IndexMap, without enabling serde_json/preserve_order. The existing spec/protobuf version is unchanged; legacy raw-type conversion helpers are retained.
- DataFusion fork b66dca326 preserves physical EXPLAIN JSON field and metric ordering through renderer-local serialization, including duplicate metric overwrite behavior, without the global feature.
With the published Git pins, the locked cmd + vector_index normal/build dependency graph has no serde_json/preserve_order. General DB JSON ordering returns to the pre-upgrade main behavior; SQLness regenerated the SST meta_json expectation accordingly. Published-pin build and 787 Substrait/datasource/query tests passed (2 skipped); equivalent local-source integration passed 44 SQLness executions. Physical pgjson tests pass with and without preserve_order. New-head CI/compatibility is still pending.
We also checked the targeted byte-sensitive paths: table-info/table-route/table-repartition CAS compares captured original KV bytes, and manifest checksums cover original payload bytes. That is a focused audit, not a claim that every JSON consumer was exhaustively audited. The PR description now documents the local-ordering decision and validation. Leaving this thread open for your review.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
…pr8555-ci-rebase Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
5 participants
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
apache/datafusion@d5552342LIMITacross multiple partitions: apache/datafusion#23800GreptimeTeam/datafusion-postgres@3c77e6c3GreptimeTeam/sqlparser-rs@9e9019bbWhat's changed and what's your intention?
This PR upgrades GreptimeDB from the DataFusion 53.1 dependency line to DataFusion
55.0.0, Arrow59.2, and Parquet59.2. The branch now merges main at38577c8854cb76bd3783e3b57aec310496d82c10without rewriting its existing history; the review-fix head is400bc00d647d3f94b912d9f19bb28b443f555041.The upgrade adapts GreptimeDB planning and execution integrations to DataFusion 55 while retaining the behaviors that are material to GreptimeDB: distributed ordering via
EnforceSorting, custom physical plans and expression traversal, recursiveMergeScanfallback handling, global Top-K throughMergeSort, table-function planning, PostgreSQL compatibility, generated range-query validation,count_hashcompatibility, JSON field ordering, and topology-specific SQLness plan expectations.Dependency versions and pins
55.0.0; forkb66dca3260c8e04f314c8036cdfbb08f1a564d9559.2.0datafusion-orc0.10.0orc-rust0.9.0datafusion-pg-catalog/arrow-pg3c77e6c32b8dsqlparser0.62.0at9e9019bbAll 15 DataFusion entries in
[patch.crates-io]use the same immutable fork revision,b66dca3260c8e04f314c8036cdfbb08f1a564d95; the corresponding 35Cargo.locksource blocks use that same revision. The released ORC packages replace the temporary ORC git patches, anddatafusion-pg-catalogandarrow-pgremain pinned together at their DataFusion 55-compatible commit.Thin-fork history
The fork starts at official DataFusion
55.0.0commitd555234201. The six original retained commits remain for these purposes:LIMIT96625f97aad1dd7b9ad6da6529d542e47063d1e46922ddce3dec6af133dcount_hashstate compatibilityThe scalar-planner hook restoration
aab310aac36is retained for JSON2. Four additional fork commits are now pushed:72872aedb(lint-only timestamp conversion cleanup),a17899ec6(recursive-CTE schema metadata),85fc18b51(Date32 predicate widening), and7ac19906c(cross-timezone regression correction).Additional retained behaviors after the old-fork audit
The final fork contains 20 commits beyond official DataFusion 55. The old DataFusion 53.1 fork was checked commit-by-commit (23 non-merge commits), including behaviors already supplied upstream rather than only matching patch titles. Five follow-up commits restore six missing repair topics:
05fe011bd768fac3de7184e55f0c6be2b6246f37579d2c9b9aAlready-upstream retention includes finer timestamp coercion and safe overflow handling, preserved-side join dynamic filters, aggregate filter ordering, SPM statistics/cardinality, dictionary-regex NULL handling, and join-estimation overflow protection. Aligned timestamp widening equality/IN is restored under the explicitly accepted extreme-range overflow policy; unsafe dictionary casts remain restricted. Ordered timestamp widening retains its inherited overflow policy, not full-domain CAST/TRY_CAST equivalence.
DB-local sorting, opaque MergeSort global Top-K handling, and post-optimizer Filter-fetch repairs remain separate from the fork patches.
count_hashpreservation requires both the DB's fixed-seed custom-hasher API and the fork's raw floating-point representation; this is not a claim that all DataFusion hashing APIs are interchangeable.Targeted repairs
JSON2 scalar-function planning
ExprPlannerhook GreptimeDB uses to supply JSON2 extraction types. Removing the obsolete GreptimeDB method would compile but makes expressions such asabs(j.metrics.value)fail during planning before analyzer rules can repair them.RawScalarExpr/plan_scalarextension point after scalar argument resolution. ModifiedOriginalarguments continue through later planners and normal function construction. GreptimeDB keeps its production JSON2 type injection; this does not introduce a test-only path or a new rewrite framework.PromQL temporality join-key encoding
otlp_aggregation_temporality, the planner supplies a NULL marker so that it matches cumulative (NULL) rows but not delta rows. Remote planning can decode this synthetic literal to plain UTF8 while the stored marker remains dictionary-encoded. DataFusion 55's multi-key hash-join comparator rejects those mixed array types, breaking arithmetic andANDin the existingdelta_temporalitySQLness case.ffba905e4cddecodes only the present dictionary marker to its string value type and supplies the missing marker as a typed NULL of that same type. Qualified column names, NULL matching, and the left operand's visible columns and values forAND/UNLESSare preserved; padding is not exposed for a left operand lacking the temporality marker. This is a local PromQL alignment fix, not a global storage conversion or another fork patch. Tests cover both operand orders, both dictionary NULL representations, exact arithmetic/set results, and a nested consumer. The original stored-primary-key SQLness expectations remain unchanged.Recursive CTE schema metadata
information_schema.tablesalso exposes a distinct DataFusion 55 physical-planning failure:Cannot project plan to expected schema: schema metadata differ. A literal seed and table-backed recursive branch may have different top-level metadata even though the declared recursive output takes the seed metadata; the query must continue to return depths0, 1, 2.a17899ec6makes the existingproject_plan_to_schemahelper constructProjectionExecwith the declared output schema metadata rather than reject only a top-level metadata difference. The exact-schema fast path remains. Column-count, data-type, nullability-narrowing, and field-metadata mismatches remain rejected. It neither strips scan metadata nor changes fix: re-scan stream-backed tables in recursive CTEs #9039's stream-reopening behavior.Date32 predicate optimization restoration
Date32 → Date64, so existingsimplify_expr.sltEXPLAIN cases could not eliminate those casts. Everyi32day value scaled by 86,400,000 fits ini64.85fc18b51allows onlyDate32 → Date64. Inverse-literal whole-day alignment and Date32-domain checks remain, andDate64 → Date32remains rejected. Coverage includes boundaries, ordered comparisons, nonaligned and out-of-range literals, and reverse-direction rejection.Cross-timezone equality regression
2024-02-01 00:00:01 +01and Los Angeles2024-01-31 16:00:01 -08as equal, although they differ by one hour. Because the view explicitly casts the input totimestampbefore assigning Brussels, its matching Los Angeles wall clock is15:00:01.7ac19906cchanges the positive literal to 15:00, retains the expected positive row, and preserves 16:00 as a no-match regression. It changes neither timezone implementation nor the positive result semantics.The lint-only commit
72872aedbremoves four redundanti128::from(timestamp_unit_scale(...))conversions: the helper already returnsi128, so strict clippy rejects these identity conversions. Timestamp arithmetic and guards are unchanged.EnforceSortingremains conservatively preserved to retain the existing GreptimeDB sorting pass and distributed plan shaping during this upgrade. This does not establish that an extra pass is intrinsically required beyond DataFusion’s defaultEnsureRequirements.User-visible changes and compatibility limits
This upgrade is not a blanket SQL/API compatibility guarantee:
median,approx_median, andapprox_percentile_contproduce floating-point results. Exact median interpolates even-sized inputs (for example,4becomes4.5in the existing regression); approximate quantile values also change. Consumers with fixed output schemas, including views and Flow sinks, need compatibility assessment; those schema migrations have not been exhaustively tested here.REPLACE(s, '', replacement)now returnssunchanged instead of interleaving the replacement string.unnestoutput column names drop the__unnest_placeholder(...)wrapper. Clients depending on generated names should use explicit aliases.approx_distinctcase changes from2000to1991. This does not promise stable estimates across versions.++; this is a display change, not added result rows.OID,REGCLASS,REGTYPE, and related types) instead of reporting only their former INT4/VARCHAR representations.serde_json/preserve_orderis no longer enabled in the verified production graph. Both sources were removed: Substrait named-struct ordinals use a local IndexMap, and physical EXPLAIN JSON uses local ordered serialization. General DB JSON object ordering remains consistent with pre-upgrade main; SST metadata expectations were regenerated accordingly. The focused raw-byte CAS/checksum audit found no issue in the inspected paths, but is not a universal byte-stability claim.Timestamp precision-widening equality and IN preimages are restored by fork commit
605effa378, reusing the existing literal conversion and round-trip alignment checks. Same-timezone widening is eligible; narrowing, nonaligned literals and unsafe dictionary casts remain guarded. The explicitly accepted policy favors normal-range time pruning even when rewriting changes CAST/TRY_CAST overflow behavior at extreme timestamps; no full-domain equivalence is claimed. Date32-to-Date64 remains eligible as well.Standalone and distributed SQLness both verify
ts = TimestampMillisecond(5000)reaches scan filters without CAST; aligned IN becomes source-column equalities. Exact/nonexact normal-range results and the accepted extreme-range policy are covered. Directarrow_try_castreturns NULL on the tested positive overflow. The pre-existing SQLTRY_CAST(... AS TIMESTAMP(9))type-alias-lowering limitation remains outside this upgrade's scope.Substrait 0.63 is patched via
GreptimeTeam/substrait-rsrevision91ec978b0649417ad3da8390e7a515baec723b1b. Its named-struct field order remains explicit and local; no protobuf/spec upgrade or DB-wide ordered-JSON abstraction is introduced. The raw Type::Object payload is now IndexMap (a Rust source API adjustment); legacy conversion helpers are retained. The fork also supports Cargo tagless checkout of the exact pinned spec release.Review follow-ups
Signature::uniformcontracts for clamp/min/max, modulo, rate and anomaly functions. Decimal inputs follow the legacy Float64 coercion path instead of reaching unsupported native-decimal execution. Actual Decimal128 coercion/execution regressions and standalone/distributed SQL column queries cover clamp and modulo. This does not add native Decimal arithmetic support.count_hashtyped-NULL behavior, including its historical zero hash contribution. Preserve fixed seeds, raw floating-point hashing and state encoding; a persisted[0]state regression checks state/result agreement with fresh typed-NULL input. The pre-existing scalar/group NULL inconsistency is deliberately not corrected in this upgrade.attnum/column-name/type associations in ordinal order.Validation and PR state
Historical timestamp-restoration validation (
400bc00d647): fork605effa378passes 9 expr-common, 12 logical optimizer and 58 physical simplifier tests plus strict three-crate all-target/all-feature Clippy. Greptime vector-index build and formatting pass; query test result:Summary [ 4.896s] 739 tests run: 739 passed, 2 skipped. Both deployment cast-preimage SQLness reruns pass with explicit bare-millisecond EQ/IN scan predicates. At head400bc00d647, Rust CI passed and Integration CI run 34457426139 attempt 2 passed, including actually executed recent-release compatibility. Logical-pruning fuzz initially failed after a candidate datanode health-probe timeout/restart, then passed on retry; the underlying stall is not established. Cargo.lock Diff Check failed while posting its size warning because the integration lacked comment permissions, not because dependency resolution failed.Earlier review-fix head
dcaedac52cd:cargo nextest run --locked -p common-functionpasses 343 tests, including Decimal coercion/execution and historical count_hash state restoration. Strict all-target Clippy forcommon-functionandcatalog, the vector-index-enabled greptime build, and workspace Rust formatting pass. Actual Decimal-column SQL queries produce the expected clamp/min/max/modulo results in standalone and distributed deployments; the generated outputs are identical and were rerun successfully.Review SQLness validation passes in both deployments: explicit sorting/column-ordinal assertions, SST metadata output, and timestamp safe/overflow boundary cases. Relocated TQL fixtures were additionally rerun at their final paths. That earlier forced-feature snapshot check is historical; current SST output is verified without global preservation. Expected files were generated by SQLness, not hand-edited.
Earlier merge validation (
a5aaebe5cc4/4321681d53f): vector-index cmd check, query/table/promql tests (1038 passed, 2 skipped), and both binary builds pass. Nineteen merge-sensitive SQLness cases have passing results after regenerating the two new Arrow59 empty-table renderings. CI on4321681d53fpassed all executed build, Rust, Clippy, SQLness, SDK and Fuzz jobs; recent-releases compatibility was skipped on synchronize. The Cargo.lock advisory job failed while attempting to post its large-diff warning (Resource not accessible by integration), not while resolving dependencies.Earlier retention-pin validation (
e749334f0d4): query tests 693 passed, 2 skipped, and 38 targeted SQLness executions passed. Recent-releases compatibility actually executed and passed in Integration run 34350572598, attempt 2. This is historical evidence, not a compatibility pass for the current review-fix head.Fork
579d2c9b9a: combined validation passes 569 physical-optimizer tests, 82 group-value tests, 57 Substrait unit tests, 202 Substrait integration tests (6 ignored), 3 doctests, and 31 coercion tests. Affected crates pass strict all-target/all-feature Clippy. Dictionary-focused normal and forced-collision runs each pass 79 tests. Native before/after regressions confirm LIMIT 1 and LIMIT 0 survive repeated optimization and ordered TopK retains[10,20]rather than four rows.Historical head
8b12b1b67ae: Rust CI 34475382126 and Integration CI 34480886694 passed, including actually executed recent-releases compatibility. The Cargo.lock warning-comment check remains red due to GitHub API comment permissions, not dependency resolution or test failure. This PR remains Ready for review and has not been merged.PR Checklist
SQL23 correctness follow-up
The fork now includes
ca00e5c7f, backporting merged Apache DataFusion #24817 (b3cb365dd6e5e0441f3f58817fee55dae1814fdb). Aggregate dynamic filtering is disabled for the entire aggregate if any MIN/MAX argument is unsupported; filtering only for the supported subset can discard rows required by expressions such asMAX(ABS(v)). All-direct-column MIN/MAX OR filtering remains enabled.Validation of this fork backport: the complete
push_down_filter_regression.sltpasses, including deterministic mixed-aggregate numeric-result and no-filter assertions; formatting and full-workspacecargo clippy --all-targets --all-features -- -D warningspass. Independent review approved the upstream backport. These are fork-level checks. DB validation is recorded below: SQL23 HTTP verification at5db4efe77f4, and successful integration/compatibility CI at current head8b12b1b67ae.The separate review of 384 upstream commits from DF55 to frozen upstream main
2db0ec3bbedb72e7fc68938feef4b4b70efa7f93is still selecting applicable fixes. The three expression fixes listed below are now included; other selected candidates remain pending and are not part of this update.NULL and predicate simplification follow-up
Fork commit
ad5c4be23backports three merged upstream correctness fixes:col ~ '.*'.Verification: 759 optimizer unit tests pass; complete
scalar.slt,simplify_expr.slt, andsimplify_predicates.sltpass; formatting and full-workspace all-target/all-feature Clippy with-D warningspass. Independent review confirmed upstream fidelity. Stale timestamp SLT expectations were aligned with the already accepted widening policy; production timestamp behavior was not changed. These are fork-level results. At Greptime head8b12b1b67ae, Rust CI run 34475382126 and Integration CI run 34480886694 passed, including actually executed recent-release compatibility (not skipped), SQLness, fuzz/chaos, SDK and export/import checks. The remaining Cargo.lock Diff Check failure is its warning-comment API permission error (Resource not accessible by integration), not dependency resolution or a test failure.Additional SQL23 DB verification at preceding head
5db4efe77f4: vector-index build passed, query tests 739 passed / 2 skipped, and the real HTTP reproducer returned the correct mixed aggregate[1000000, 1700000019999]in all 20 repetitions, with no incomplete aggregate dynamic filter in EXPLAIN ANALYZE. This is not a claim of DB verification for the newer expression-fix pin.The selected Parquet statistics, DISTINCT/fetch/OFFSET, legacy aggregate spill, and window-argument fixes have not yet been backported and are not included.
Current merge and JSON-order isolation —
bc7ecdce88fMerged main
9dabe8ccd232739614b21119bc63116567538653into this branch (no rebase). Kept main’s proto/jemallocator updates and SQL23 regression tests, retained DF55, and pinned DFb66dca326plus Substrait91ec978. No additional deferred upstream fixes are included.Validation: published Git pins resolve and build; locked
cmd + vector_indexnormal/build feature graph contains noserde_json/preserve_order; DB Substrait/datasource/query tests 787 passed, 2 skipped. Equivalent local-source integration ran 44 actual SQLness executions across standalone/distributed JSON, SST, SQL23 and applicable copy cases, then reran green after generated SST key-order expectations returned to the main baseline. Physical pgjson tests pass in both feature states and full DataFusion strict Clippy passes; Substrait has 59 unit + 3 doc tests passing.New-head CI and recent-release compatibility are pending. Previous-head CI results above are historical, not validation of this merge. The unresolved JSON review thread has not been automatically resolved or replied to.